{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas 排序性能对比和最佳实践\n\n本教程详细对比各种排序方法的性能，并提供最佳实践指导。\n\n## 目录\n1. 排序方法性能对比\n2. 大数据集排序优化\n3. 内存使用优化\n4. 排序算法选择\n5. 最佳实践总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\nimport numpy as np\nimport time\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nplt.rcParams['font.sans-serif'] = ['SimHei']\nplt.rcParams['axes.unicode_minus'] = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 排序方法性能对比\n\n### 方法说明\n\n对比不同排序方法的性能特点：\n- `sort_values()`: 完整排序\n- `sort_index()`: 索引排序  \n- `nlargest()/nsmallest()`: 获取极值\n- `numpy.argsort()`: NumPy排序\n\n**测试场景:**\n- 不同数据规模\n- 不同数据类型\n- 不同排序需求"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 性能测试1: 获取前N名的不同方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def performance_test_topn():\n    sizes = [1000, 5000, 10000, 50000]\n    methods = ['nlargest', 'sort_values+head', 'numpy_argsort']\n    results = {method: [] for method in methods}\n    \n    for size in sizes:\n        df = pd.DataFrame({\n            'value': np.random.randint(1, 10000, size),\n            'index': range(size)\n        })\n        \n        n = 10  # 获取前10名\n        \n        # 方法1: nlargest\n        start = time.time()\n        result1 = df.nlargest(n, 'value')\n        results['nlargest'].append(time.time() - start)\n        \n        # 方法2: sort_values + head\n        start = time.time()\n        result2 = df.sort_values('value', ascending=False).head(n)\n        results['sort_values+head'].append(time.time() - start)\n        \n        # 方法3: numpy argsort\n        start = time.time()\n        indices = np.argsort(df['value'].values)[-n:][::-1]\n        result3 = df.iloc[indices]\n        results['numpy_argsort'].append(time.time() - start)\n    \n    return sizes, results\n\nsizes, results = performance_test_topn()\n\nprint(\"获取前10名的性能对比:\")\nfor i, size in enumerate(sizes):\n    print(f\"\\n数据规模: {size:,} 行\")\n    for method, times in results.items():\n        print(f\"  {method:20}: {times[i]:.6f} 秒\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 性能测试2: 完整排序vs部分排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_full_vs_partial():\n    df = pd.DataFrame({\n        'value': np.random.randint(1, 100000, 20000)\n    })\n    \n    # 完整排序\n    start = time.time()\n    full_sort = df.sort_values('value')\n    time_full = time.time() - start\n    \n    # 部分排序（前100名）\n    start = time.time()\n    partial_sort = df.nlargest(100, 'value')\n    time_partial = time.time() - start\n    \n    print(\"完整排序 vs 部分排序 (20000行数据):\")\n    print(f\"完整排序:     {time_full:.6f} 秒\")\n    print(f\"部分排序(100): {time_partial:.6f} 秒\")\n    print(f\"部分排序快 {time_full/time_partial:.1f} 倍\")\n\ncompare_full_vs_partial()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 大数据集排序优化\n\n### 优化策略\n\n处理大数据集时的优化方法：\n1. 数据类型优化\n2. 分块处理\n3. 索引优化\n4. 内存管理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化1: 数据类型优化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_dtype_optimization():\n    size = 100000\n    \n    # 创建不同数据类型的数据\n    df_int64 = pd.DataFrame({'value': np.random.randint(1, 1000, size, dtype='int64')})\n    df_int32 = pd.DataFrame({'value': np.random.randint(1, 1000, size, dtype='int32')})\n    df_int16 = pd.DataFrame({'value': np.random.randint(1, 1000, size, dtype='int16')})\n    \n    datasets = [\n        ('int64', df_int64),\n        ('int32', df_int32), \n        ('int16', df_int16)\n    ]\n    \n    print(\"不同数据类型的排序性能:\")\n    for dtype_name, df in datasets:\n        start = time.time()\n        df.sort_values('value')\n        sort_time = time.time() - start\n        memory = df.memory_usage(deep=True).sum() / 1024 / 1024\n        print(f\"{dtype_name:6}: {sort_time:.6f}秒, 内存: {memory:.2f}MB\")\n\ntest_dtype_optimization()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化2: 分块处理大数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chunked_sort_demo():\n    # 模拟大数据集分块排序\n    total_size = 50000\n    chunk_size = 10000\n    \n    # 创建大数据集\n    large_data = np.random.randint(1, 100000, total_size)\n    \n    # 分块排序\n    start = time.time()\n    sorted_chunks = []\n    for i in range(0, total_size, chunk_size):\n        chunk = large_data[i:i+chunk_size]\n        sorted_chunk = np.sort(chunk)\n        sorted_chunks.append(sorted_chunk)\n    \n    # 合并排序结果（简化版）\n    final_result = np.concatenate(sorted_chunks)\n    chunk_time = time.time() - start\n    \n    # 直接排序对比\n    start = time.time()\n    direct_sort = np.sort(large_data)\n    direct_time = time.time() - start\n    \n    print(\"分块排序 vs 直接排序:\")\n    print(f\"分块排序: {chunk_time:.6f} 秒\")\n    print(f\"直接排序: {direct_time:.6f} 秒\")\n\nchunked_sort_demo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 内存使用优化\n\n### 内存优化策略\n\n减少排序时的内存使用：\n1. inplace操作\n2. 删除不需要的列\n3. 使用视图而非副本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 内存测试: inplace vs 副本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import psutil\nimport os\n\ndef get_memory_usage():\n    process = psutil.Process(os.getpid())\n    return process.memory_info().rss / 1024 / 1024  # MB\n\ndef memory_comparison():\n    df = pd.DataFrame({\n        'col1': np.random.randint(1, 1000, 50000),\n        'col2': np.random.randint(1, 1000, 50000),\n        'col3': np.random.randint(1, 1000, 50000)\n    })\n    \n    initial_memory = get_memory_usage()\n    \n    # 方法1: 创建副本\n    start_memory = get_memory_usage()\n    df_copy = df.sort_values('col1')\n    copy_memory = get_memory_usage()\n    \n    # 方法2: inplace操作\n    df_test = df.copy()\n    start_memory2 = get_memory_usage()\n    df_test.sort_values('col1', inplace=True)\n    inplace_memory = get_memory_usage()\n    \n    print(\"内存使用对比:\")\n    print(f\"初始内存: {initial_memory:.1f} MB\")\n    print(f\"副本排序: +{copy_memory - start_memory:.1f} MB\")\n    print(f\"inplace排序: +{inplace_memory - start_memory2:.1f} MB\")\n\nmemory_comparison()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 排序算法选择\n\n### 算法对比\n\n不同排序算法的特点：\n- quicksort: 平均最快\n- mergesort: 稳定排序\n- heapsort: 最坏情况性能好"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 算法性能测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def algorithm_comparison():\n    algorithms = ['quicksort', 'mergesort', 'heapsort']\n    sizes = [1000, 5000, 10000]\n    \n    results = {}\n    \n    for size in sizes:\n        df = pd.DataFrame({'value': np.random.randint(1, 10000, size)})\n        results[size] = {}\n        \n        for alg in algorithms:\n            start = time.time()\n            df.sort_values('value', kind=alg)\n            results[size][alg] = time.time() - start\n    \n    print(\"不同算法性能对比:\")\n    for size in sizes:\n        print(f\"\\n数据规模: {size:,} 行\")\n        for alg in algorithms:\n            print(f\"  {alg:10}: {results[size][alg]:.6f} 秒\")\n\nalgorithm_comparison()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 最佳实践总结\n\n### 实践指南\n\n根据不同场景选择最优方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 场景1: 获取Top-N数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 推荐方法对比\ndf = pd.DataFrame({\n    'sales': np.random.randint(1000, 50000, 10000),\n    'region': np.random.choice(['North', 'South', 'East', 'West'], 10000)\n})\n\n# ✅ 推荐: nlargest\ntop10 = df.nlargest(10, 'sales')\n\n# ❌ 不推荐: sort + head (对大数据集)\n# top10_slow = df.sort_values('sales', ascending=False).head(10)\n\nprint(\"Top-N 推荐方法:\")\nprint(\"✅ df.nlargest(n, 'column')\")\nprint(\"❌ df.sort_values('column').head(n)  # 大数据集时慢\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 场景2: 多列排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 多列排序最佳实践\ndf_multi = pd.DataFrame({\n    'category': np.random.choice(['A', 'B', 'C'], 1000),\n    'priority': np.random.choice([1, 2, 3], 1000),\n    'value': np.random.randint(1, 100, 1000)\n})\n\n# ✅ 推荐: sort_values 多列\nsorted_df = df_multi.sort_values(['category', 'priority', 'value'], \n                                ascending=[True, True, False])\n\nprint(\"多列排序推荐:\")\nprint(\"✅ df.sort_values(['col1', 'col2'], ascending=[True, False])\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 场景3: 时间序列排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 时间序列排序\ndates = pd.date_range('2023-01-01', periods=1000, freq='H')\nts_df = pd.DataFrame({\n    'timestamp': np.random.choice(dates, 1000),\n    'value': np.random.randn(1000)\n})\n\n# ✅ 推荐: 先设置时间索引再排序\nts_df.set_index('timestamp', inplace=True)\nsorted_ts = ts_df.sort_index()\n\nprint(\"时间序列排序推荐:\")\nprint(\"✅ df.set_index('timestamp').sort_index()\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 性能优化清单"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Pandas 排序性能优化清单:\")\nprint()\nprint(\"🚀 速度优化:\")\nprint(\"  ✅ Top-N 使用 nlargest/nsmallest\")\nprint(\"  ✅ 大数据集考虑分块处理\")\nprint(\"  ✅ 选择合适的排序算法\")\nprint(\"  ✅ 预先优化数据类型\")\nprint()\nprint(\"💾 内存优化:\")\nprint(\"  ✅ 不需要原数据时使用 inplace=True\")\nprint(\"  ✅ 删除不需要的列\")\nprint(\"  ✅ 使用合适的数据类型\")\nprint()\nprint(\"📊 场景选择:\")\nprint(\"  ✅ 完整排序: sort_values()\")\nprint(\"  ✅ 索引排序: sort_index()\")\nprint(\"  ✅ 前N名: nlargest/nsmallest\")\nprint(\"  ✅ 多列排序: sort_values(['col1', 'col2'])\")\nprint()\nprint(\"⚠️  避免的做法:\")\nprint(\"  ❌ 大数据集用 sort + head 获取 Top-N\")\nprint(\"  ❌ 不必要的数据复制\")\nprint(\"  ❌ 在循环中重复排序\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}